<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
                "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
  <title>Description of demmdn1</title>
  <meta name="keywords" content="demmdn1">
  <meta name="description" content="DEMMDN1 Demonstrate fitting a multi-valued function using a Mixture Density Network.">
  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
  <meta name="generator" content="m2html &copy; 2003 Guillaume Flandin">
  <meta name="robots" content="index, follow">
  <link type="text/css" rel="stylesheet" href="../../m2html.css">
</head>
<body>
<a name="_top"></a>
<div><a href="../../menu.html">Home</a> &gt;  <a href="#">ReBEL-0.2.7</a> &gt; <a href="#">netlab</a> &gt; demmdn1.m</div>

<!--<table width="100%"><tr><td align="left"><a href="../../menu.html"><img alt="<" border="0" src="../../left.png">&nbsp;Master index</a></td>
<td align="right"><a href="menu.html">Index for .\ReBEL-0.2.7\netlab&nbsp;<img alt=">" border="0" src="../../right.png"></a></td></tr></table>-->

<h1>demmdn1
</h1>

<h2><a name="_name"></a>PURPOSE <a href="#_top"><img alt="^" border="0" src="../../up.png"></a></h2>
<div class="box"><strong>DEMMDN1 Demonstrate fitting a multi-valued function using a Mixture Density Network.</strong></div>

<h2><a name="_synopsis"></a>SYNOPSIS <a href="#_top"><img alt="^" border="0" src="../../up.png"></a></h2>
<div class="box"><strong>This is a script file. </strong></div>

<h2><a name="_description"></a>DESCRIPTION <a href="#_top"><img alt="^" border="0" src="../../up.png"></a></h2>
<div class="fragment"><pre class="comment">DEMMDN1 Demonstrate fitting a multi-valued function using a Mixture Density Network.

    Description
    The problem consists of one input variable X and one target variable
    T with data generated by sampling T at equal intervals and then
    generating target data by computing T + 0.3*SIN(2*PI*T) and adding
    Gaussian noise. A Mixture Density Network with 3 centres in the
    mixture model is trained by minimizing a negative log likelihood
    error function using the scaled conjugate gradient optimizer.

    The conditional means, mixing coefficients and variances are plotted
    as a function of X, and a contour plot of the full conditional
    density is also generated.

    See also
    <a href="mdn.html" class="code" title="function net = mdn(nin, nhidden, ncentres, dim_target, mix_type,prior, beta)">MDN</a>, <a href="mdnerr.html" class="code" title="function e = mdnerr(net, x, t)">MDNERR</a>, <a href="mdngrad.html" class="code" title="function g = mdngrad(net, x, t)">MDNGRAD</a>, <a href="scg.html" class="code" title="function [x, options, flog, pointlog, scalelog] = scg(f, x, options, gradf, varargin)">SCG</a></pre></div>

<!-- crossreference -->
<h2><a name="_cross"></a>CROSS-REFERENCE INFORMATION <a href="#_top"><img alt="^" border="0" src="../../up.png"></a></h2>
This function calls:
<ul style="list-style-image:url(../../matlabicon.gif)">
<li><a href="gmmprob.html" class="code" title="function prob = gmmprob(mix, x)">gmmprob</a>	GMMPROB Computes the data probability for a Gaussian mixture model.</li><li><a href="mdn.html" class="code" title="function net = mdn(nin, nhidden, ncentres, dim_target, mix_type,prior, beta)">mdn</a>	MDN	Creates a Mixture Density Network with specified architecture.</li><li><a href="mdn2gmm.html" class="code" title="function gmmmixes = mdn2gmm(mdnmixes)">mdn2gmm</a>	MDN2GMM Converts an MDN mixture data structure to array of GMMs.</li><li><a href="mdnfwd.html" class="code" title="function [mixparams, y, z, a] = mdnfwd(net, x)">mdnfwd</a>	MDNFWD	Forward propagation through Mixture Density Network.</li><li><a href="mdninit.html" class="code" title="function net = mdninit(net, prior, t, options)">mdninit</a>	MDNINIT Initialise the weights in a Mixture Density Network.</li><li><a href="mlp.html" class="code" title="function net = mlp(nin, nhidden, nout, outfunc, prior, beta)">mlp</a>	MLP	Create a 2-layer feedforward network.</li><li><a href="mlpfwd.html" class="code" title="function [y, z, a] = mlpfwd(net, x)">mlpfwd</a>	MLPFWD	Forward propagation through 2-layer network.</li><li><a href="netopt.html" class="code" title="function [net, options, varargout] = netopt(net, options, x, t, alg);">netopt</a>	NETOPT	Optimize the weights in a network model.</li></ul>
This function is called by:
<ul style="list-style-image:url(../../matlabicon.gif)">
<li><a href="demnlab.html" class="code" title="function demnlab(action);">demnlab</a>	DEMNLAB A front-end Graphical User Interface to the demos</li></ul>
<!-- crossreference -->


<h2><a name="_source"></a>SOURCE CODE <a href="#_top"><img alt="^" border="0" src="../../up.png"></a></h2>
<div class="fragment"><pre>0001 <span class="comment">%DEMMDN1 Demonstrate fitting a multi-valued function using a Mixture Density Network.</span>
0002 <span class="comment">%</span>
0003 <span class="comment">%    Description</span>
0004 <span class="comment">%    The problem consists of one input variable X and one target variable</span>
0005 <span class="comment">%    T with data generated by sampling T at equal intervals and then</span>
0006 <span class="comment">%    generating target data by computing T + 0.3*SIN(2*PI*T) and adding</span>
0007 <span class="comment">%    Gaussian noise. A Mixture Density Network with 3 centres in the</span>
0008 <span class="comment">%    mixture model is trained by minimizing a negative log likelihood</span>
0009 <span class="comment">%    error function using the scaled conjugate gradient optimizer.</span>
0010 <span class="comment">%</span>
0011 <span class="comment">%    The conditional means, mixing coefficients and variances are plotted</span>
0012 <span class="comment">%    as a function of X, and a contour plot of the full conditional</span>
0013 <span class="comment">%    density is also generated.</span>
0014 <span class="comment">%</span>
0015 <span class="comment">%    See also</span>
0016 <span class="comment">%    MDN, MDNERR, MDNGRAD, SCG</span>
0017 <span class="comment">%</span>
0018 
0019 <span class="comment">%    Copyright (c) Ian T Nabney (1996-2001)</span>
0020 
0021 
0022 <span class="comment">% Generate the matrix of inputs x and targets t.</span>
0023 seedn = 42;
0024 seed = 42;
0025 randn(<span class="string">'state'</span>, seedn);
0026 rand(<span class="string">'state'</span>, seed);
0027 ndata = 300;            <span class="comment">% Number of data points.</span>
0028 noise = 0.2;            <span class="comment">% Range of noise distribution.</span>
0029 t = [0:1/(ndata - 1):1]';
0030 x = t + 0.3*sin(2*pi*t) + noise*rand(ndata, 1) - noise/2;
0031 axis_limits = [-0.2 1.2 -0.2 1.2];
0032 
0033 clc
0034 disp(<span class="string">'This demonstration illustrates the use of a Mixture Density Network'</span>)
0035 disp(<span class="string">'to model multi-valued functions.  The data is generated from the'</span>)
0036 disp(<span class="string">'mapping x = t + 0.3 sin(2 pi t) + e, where e is a noise term.'</span>)
0037 disp(<span class="string">'We begin by plotting the data.'</span>)
0038 disp(<span class="string">' '</span>)
0039 disp(<span class="string">'Press any key to continue'</span>)
0040 pause
0041 <span class="comment">% Plot the data</span>
0042 fh1 = figure;
0043 p1 = plot(x, t, <span class="string">'ob'</span>);
0044 axis(axis_limits);
0045 hold on
0046 disp(<span class="string">'Note that for x in the range 0.35 to 0.65, there are three possible'</span>)
0047 disp(<span class="string">'branches of the function.'</span>)
0048 disp(<span class="string">' '</span>)
0049 disp(<span class="string">'Press any key to continue'</span>)
0050 pause
0051 
0052 <span class="comment">% Set up network parameters.</span>
0053 nin = 1;            <span class="comment">% Number of inputs.</span>
0054 nhidden = 5;            <span class="comment">% Number of hidden units.</span>
0055 ncentres = 3;            <span class="comment">% Number of mixture components.</span>
0056 dim_target = 1;            <span class="comment">% Dimension of target space</span>
0057 mdntype = <span class="string">'0'</span>;            <span class="comment">% Currently unused: reserved for future use</span>
0058 alpha = 100;            <span class="comment">% Inverse variance for weight initialisation</span>
0059                 <span class="comment">% Make variance small for good starting point</span>
0060 
0061 <span class="comment">% Create and initialize network weight vector.</span>
0062 net = <a href="mdn.html" class="code" title="function net = mdn(nin, nhidden, ncentres, dim_target, mix_type,prior, beta)">mdn</a>(nin, nhidden, ncentres, dim_target, mdntype);
0063 init_options = zeros(1, 18);
0064 init_options(1) = -1;    <span class="comment">% Suppress all messages</span>
0065 init_options(14) = 10;  <span class="comment">% 10 iterations of K means in gmminit</span>
0066 net = <a href="mdninit.html" class="code" title="function net = mdninit(net, prior, t, options)">mdninit</a>(net, alpha, t, init_options);
0067 
0068 <span class="comment">% Set up vector of options for the optimiser.</span>
0069 options = foptions;
0070 options(1) = 1;            <span class="comment">% This provides display of error values.</span>
0071 options(14) = 200;        <span class="comment">% Number of training cycles.</span>
0072 
0073 clc
0074 disp(<span class="string">'We initialise the neural network model, which is an MLP with a'</span>)
0075 disp(<span class="string">'Gaussian mixture model with three components and spherical variance'</span>)
0076 disp(<span class="string">'as the error function.  This enables us to model the complete'</span>)
0077 disp(<span class="string">'conditional density function.'</span>)
0078 disp(<span class="string">' '</span>)
0079 disp(<span class="string">'Next we train the model for 200 epochs using a scaled conjugate gradient'</span>)
0080 disp(<span class="string">'optimizer.  The error function is the negative log likelihood of the'</span>)
0081 disp(<span class="string">'training data.'</span>)
0082 disp(<span class="string">' '</span>)
0083 disp(<span class="string">'Press any key to continue.'</span>)
0084 pause
0085 
0086 <span class="comment">% Train using scaled conjugate gradients.</span>
0087 [net, options] = <a href="netopt.html" class="code" title="function [net, options, varargout] = netopt(net, options, x, t, alg);">netopt</a>(net, options, x, t, <span class="string">'scg'</span>);
0088 
0089 disp(<span class="string">' '</span>)
0090 disp(<span class="string">'Press any key to continue.'</span>)
0091 pause
0092 
0093 clc
0094 disp(<span class="string">'We can also train a conventional MLP with sum of squares error function.'</span>)
0095 disp(<span class="string">'This will approximate the conditional mean, which is not always a'</span>)
0096 disp(<span class="string">'good representation of the data.  Note that the error function is the'</span>)
0097 disp(<span class="string">'sum of squares error on the training data, which accounts for the'</span>)
0098 disp(<span class="string">'different values from training the MDN.'</span>)
0099 disp(<span class="string">' '</span>)
0100 disp(<span class="string">'We train the network with the quasi-Newton optimizer for 80 epochs.'</span>)
0101 disp(<span class="string">' '</span>)
0102 disp(<span class="string">'Press any key to continue.'</span>)
0103 pause
0104 mlp_nhidden = 8;
0105 net2 = <a href="mlp.html" class="code" title="function net = mlp(nin, nhidden, nout, outfunc, prior, beta)">mlp</a>(nin, mlp_nhidden, dim_target, <span class="string">'linear'</span>);
0106 options(14) = 80; 
0107 [net2, options] = <a href="netopt.html" class="code" title="function [net, options, varargout] = netopt(net, options, x, t, alg);">netopt</a>(net2, options, x, t, <span class="string">'quasinew'</span>);
0108 disp(<span class="string">' '</span>)
0109 disp(<span class="string">'Press any key to continue.'</span>)
0110 pause
0111 
0112 clc
0113 disp(<span class="string">'Now we plot the underlying function, the MDN prediction,'</span>)
0114 disp(<span class="string">'represented by the mode of the conditional distribution, and the'</span>)
0115 disp(<span class="string">'prediction of the conventional MLP.'</span>)
0116 disp(<span class="string">' '</span>)
0117 disp(<span class="string">'Press any key to continue.'</span>)
0118 pause
0119 
0120 <span class="comment">% Plot the original function, and the trained network function.</span>
0121 plotvals = [0:0.01:1]';
0122 mixes = <a href="mdn2gmm.html" class="code" title="function gmmmixes = mdn2gmm(mdnmixes)">mdn2gmm</a>(<a href="mdnfwd.html" class="code" title="function [mixparams, y, z, a] = mdnfwd(net, x)">mdnfwd</a>(net, plotvals));
0123 axis(axis_limits);
0124 yplot = t+0.3*sin(2*pi*t);
0125 p2 = plot(yplot, t, <span class="string">'--y'</span>);
0126 
0127 <span class="comment">% Use the mode to represent the function</span>
0128 y = zeros(1, length(plotvals));
0129 priors = zeros(length(plotvals), ncentres);
0130 c = zeros(length(plotvals), 3);
0131 widths = zeros(length(plotvals), ncentres);
0132 <span class="keyword">for</span> i = 1:length(plotvals)
0133   [m, j] = max(mixes(i).priors);
0134   y(i) = mixes(i).centres(j,:);
0135   c(i,:) = mixes(i).centres';
0136 <span class="keyword">end</span>
0137 p3 = plot(plotvals, y, <span class="string">'*r'</span>);
0138 p4 = plot(plotvals, <a href="mlpfwd.html" class="code" title="function [y, z, a] = mlpfwd(net, x)">mlpfwd</a>(net2, plotvals), <span class="string">'g'</span>);
0139 set(p4, <span class="string">'LineWidth'</span>, 2);
0140 legend([p1 p2 p3 p4], <span class="string">'data'</span>, <span class="string">'function'</span>, <span class="string">'MDN mode'</span>, <span class="string">'MLP mean'</span>, 4);
0141 hold off
0142 
0143 clc
0144 disp(<span class="string">'We can also plot how the mixture model parameters depend on x.'</span>)
0145 disp(<span class="string">'First we plot the mixture centres, then the priors and finally'</span>)
0146 disp(<span class="string">'the variances.'</span>)
0147 disp(<span class="string">' '</span>)
0148 disp(<span class="string">'Press any key to continue.'</span>)
0149 pause
0150 fh2 = figure;
0151 subplot(3, 1, 1)
0152 plot(plotvals, c)
0153 hold on
0154 title(<span class="string">'Mixture centres'</span>)
0155 legend(<span class="string">'centre 1'</span>, <span class="string">'centre 2'</span>, <span class="string">'centre 3'</span>)
0156 hold off
0157 
0158 priors = reshape([mixes.priors], mixes(1).ncentres, size(mixes, 2))';
0159 <span class="comment">%%fh3 = figure;</span>
0160 subplot(3, 1, 2)
0161 plot(plotvals, priors)
0162 hold on
0163 title(<span class="string">'Mixture priors'</span>)
0164 legend(<span class="string">'centre 1'</span>, <span class="string">'centre 2'</span>, <span class="string">'centre 3'</span>)
0165 hold off
0166 
0167 variances = reshape([mixes.covars], mixes(1).ncentres, size(mixes, 2))';
0168 <span class="comment">%%fh4 = figure;</span>
0169 subplot(3, 1, 3)
0170 plot(plotvals, variances)
0171 hold on
0172 title(<span class="string">'Mixture variances'</span>)
0173 legend(<span class="string">'centre 1'</span>, <span class="string">'centre 2'</span>, <span class="string">'centre 3'</span>)
0174 hold off
0175 
0176 disp(<span class="string">'The last figure is a contour plot of the conditional probability'</span>)
0177 disp(<span class="string">'density generated by the Mixture Density Network.  Note how it'</span>)
0178 disp(<span class="string">'is well matched to the regions of high data density.'</span>)
0179 disp(<span class="string">' '</span>)
0180 disp(<span class="string">'Press any key to continue.'</span>)
0181 pause
0182 <span class="comment">% Contour plot for MDN.</span>
0183 i = 0:0.01:1.0;
0184 j = 0:0.01:1.0;
0185 
0186 [I, J] = meshgrid(i,j);
0187 I = I(:);
0188 J = J(:);
0189 li = length(i);
0190 lj = length(j);
0191 Z = zeros(li, lj);
0192 <span class="keyword">for</span> k = 1:li;
0193   Z(:,k) = <a href="gmmprob.html" class="code" title="function prob = gmmprob(mix, x)">gmmprob</a>(mixes(k), j');
0194 <span class="keyword">end</span>
0195 fh5 = figure;
0196 <span class="comment">% Set up levels by hand to make a good figure</span>
0197 v = [2 2.5 3 3.5 5:3:18];
0198 contour(i, j, Z, v)
0199 hold on
0200 title(<span class="string">'Contour plot of conditional density'</span>)
0201 hold off
0202 
0203 disp(<span class="string">' '</span>)
0204 disp(<span class="string">'Press any key to exit.'</span>)
0205 pause
0206 close(fh1);
0207 close(fh2);
0208 <span class="comment">%%close(fh3);</span>
0209 <span class="comment">%%close(fh4);</span>
0210 close(fh5);
0211 <span class="comment">%%clear all;</span></pre></div>
<hr><address>Generated on Tue 26-Sep-2006 10:36:21 by <strong><a href="http://www.artefact.tk/software/matlab/m2html/">m2html</a></strong> &copy; 2003</address>
</body>
</html>